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ABSTRACT 

This paper studies the reasons for low response rates in online evaluations. Survey data are 
collected from the students to understand factors that might affect student participation in the 
course evaluation process. When course evaluations were opened to the student body, an email 
announcement was sent to all students, and a reminder email was sent a week later. Our study 
showed that participation rates increased not only when emails were sent, but also when faculty 
used in-class time to emphasize the importance of completing the evaluations. 
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INTRODUCTION 



eacher and course evaluations are conducted at the end of each academic term. We often assume 
that students look forward to the opportunity to give their opinions about the classes they have just 
completed. In today’s environment, we also believe that students are comfortable with expressing 
their opinions online. Taking these two assumptions together, we might expect that students would always take 
advantage of the opportunity to express their opinions about their classes in an online setting. However, most 
research indicates that student response rates collected online are far lower than those collected on paper. In fact, 
Nulty (2008) reports overall results from a number of online evaluation cases and finds that the online response rate 
averaged out to a drop of 23% in comparison to the in-class collection response rate. In spite of this drop in 
response rates, institutions continue to switch to the online format for collection of course evaluation data. 
Anderson et al (2006) reported that the number of institutions using online evaluations rose from 2% in 2000 to 33% 
in 2005. 


The Quinlan School of Business at Loyola University Chicago moved from in-class to online evaluations 
for the entire set of classes that we offer at both the undergraduate and graduate levels in the spring term of 2009. 
Across all departments, there was a significant drop in the number of responses. However, analysis of the results 
showed that there was no significant difference in the paper and online values given to courses or faculty for whom 
evaluations existed in both paper and online formats (Guder et al, 2009). However, with the online evaluations, the 
response rate dropped by 26% on the undergraduate level (Guder & Malliaris, 2010). Given this significant drop, 
we were interested in not only the comparison of values with the previous years, but also why the drop might have 
occurred. 

Anderson, Brown and Spaeth (2006) briefly explored, with a small number of students, why students did 
not respond to an online evaluation and grouped the responses into the following four reasons: 1) the students were 
disengaged (that is, they forgot or were too busy), 2) they had technology problems, 3) they perceived no benefit 
from participation, or 4) “other”. Expanding upon this study, we surveyed a large number of students representative 
of both the graduate and undergraduate populations to investigate the reasons behind their level of participation in 
the course evaluation process. 

DATA 


A brief questionnaire with items about participation in the previous term’s online course evaluation system 
was distributed to both undergraduate and graduate students during class. For most questions, the students were 
asked to circle the appropriate answer. However, we also gave the students space to give two open-ended responses 
at the end. Each of the students was asked to respond to three questions about basic interaction with the survey: 1) 
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did you receive the initial email, 2) did you receive the reminder email, and 3) did your instructor encourage you to 
complete the evaluation. The course evaluation system pulls the list of student names from the official class 
registration system and all students who are officially enrolled receive the email. However, not all students 
regularly check their school email. That is, they can say they did not receive an email when they actually mean they 
did not check their university email account for the notice. Within the system that is used at Loyola, if a student has 
filled out the evaluations as a result of the initial email, no reminder is sent to them. Reminder emails go only to the 
students who have not completed all of their set of evaluations. 

Following these three initial questions, they then identified if they completed All, Some, or None of their 
evaluations. If they responded with Some or None, then they were given a set of reasons to select from for their 

non-response (forgot to do them, was too busy, worried about being identified, evaluations don’t matter, other_). 

Finally, students were asked for their opinion of how we could encourage greater response in the future. 

To help in understanding factors that might affect student participation in the course evaluation process, we 
collected data from a representative and non-overlapping group. That is, we gathered data from all classes taught at 
the same time on the same day. In the undergraduate school, all students attending classes taught on Tuesday and 
Thursdays from 11:30 to 12:45 were given surveys in class and asked to complete them if they had taken classes in 
the SBA during the previous term. On the graduate level, all classes taught on Mondays (6:00-9:00 pm) were asked 
to participate. Using this method, we were able to obtain input from students in both the core and advanced classes. 
Each student surveyed was unique ensuring no duplication or redundancy. The selection of students who took the 
survey represented 42% of all enrolled graduate students and 36% of all enrolled undergraduates. At least one class 
from each of the academic areas was included in the survey. Responses referred to only the immediately previous 
term. The number of students at each level who completed the survey are shown in Table 1. 


Table 1: Number of Students Completing the Survey 


Level 

Total 

Graduate 

341 

Undergraduate 

771 

Grand Total 

1112 


RESULTS 


Students were asked whether they had completed All, Some, or None of the previous term’s course 
evaluations. The percentage of graduate and undergraduate students in each of those categories is given in Table 2. 
The results are shown as percent of level. 


Table 2: Percentage Completing All, Some, or None of Their Evaluations 


Level 

All 

Some 

None 

Total 

Graduate 

67.45% 

13.49% 

19.06% 

100.00% 

Undergraduate 

44.10% 

41.50% 

14.40% 

100.00% 


Graduate students were more likely than undergraduates to complete all their evaluations. Typical graduate 
students, most who work full-time, however, take two or three courses per term, while the undergraduates typically 
take five or six. We further see that if a graduate student does not complete all the evaluations, they are more likely 
to do none of them rather than to do a partial set. On the undergraduate side, if a student does not do all of the 
evaluations, they are more likely to do some of them than none. A topic for further study might be the selection of 
courses they choose to evaluate if they are not doing all of them. It is also of interest to note that the percentage of 
undergraduates completing none of their evaluations is smaller than the graduate percentage. 

Within the categories of All, Some, and None, we then looked at the percentage of students who recalled 
faculty mention of the evaluation process. Was there any effect from having the teacher specifically encourage the 
students to complete the course evaluations? When a teacher mentions, in class, the importance of filling out the 
evaluations, it brings the evaluations to the student’s attention and increases the likelihood that they might look at 
the email rather than deleting without opening it. Table 3 shows that, on both the graduate and undergraduate levels. 
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the students who filled out all of the evaluations were more likely to have been encouraged by some faculty member 
to do so. 


Table 3: Effect of Teacher Encouragement 


Level 

Encouraged 

All 

Some 

None 

Total 

Graduate 

No 

60.94% 

15.63% 

23.44% 

100.00% 


Yes 

68.95% 

13.00% 

18.05% 

100.00% 

Undergraduate 

No 

31.69% 

50.00% 

18.31% 

100.00% 


Yes 

46.90% 

39.59% 

13.51% 

100.00% 


Specifically on the graduate level, for the set of students in classes where no teacher encouraged them to fill 
out the evaluations, about 61% of them completed all the evaluations anyway, while about 23% completed none of 
them. For the graduate students who had some teacher mention the importance of responding to the email, 69% 
completed all their course evaluations, while 18% did none of them. For those who completed some of the 
evaluations, the in-class mention by the professor was not beneficial. On the undergraduate side, if no professor 
encouraged them to complete their course evaluations, only around 32% of them filled all of them out, while 50% 
completed some and 18% did none. For undergraduates who had a faculty mention the importance of this process, 
47% filled out all their evaluations, almost 40% did some of them and less than 14% did none of them. Almost 47% 
of the undergraduates who were encouraged completed all their evaluations, which, compared with 32% of those 
who were not encouraged, is an increase of 15%. 

In both the graduate and undergraduate cases, the percent of students who completely disregarded the 
process by filling out none of their evaluations was smaller for those who had received encouragement by a faculty 
member than for those where no faculty member brought up the importance of the evaluations in class. 

Table 4 adds the information from the two questions of whether they remembered receiving the original 
email and the reminder. Recall that the reminder was sent only to those who had not already completed their course 
evaluations. For the graduate students, notice that they all recalled receiving the original email with the link to the 
evaluation form. The largest percentages of those who filled out all their evaluations go to those who were 
encouraged by some teacher to do so. The largest percentages of the graduate students who completed none of their 
evaluations goes to those who received no encouragement from faculty regardless of emails and reminders. 


Table 4: Responses by Level for Encouragement, Email, and Reminder 


Level 

Encouraged 

Email 

Reminder 

All 

Some 

None 

Total 

Graduate 

No 

Yes 

No 

50.00% 

25.00% 

25.00% 

100% 


No 

Yes 

Yes 

61.67% 

15.00% 

23.33% 

100% 


Yes 

Yes 

No 

72.22% 

16.67% 

11.11% 

100% 


Yes 

Yes 

Yes 

68.73% 

12.74% 

18.53% 

100% 

Undergraduate 

No 

Yes 

No 

40.00% 

46.67% 

13.33% 

100% 


No 

Yes 

Yes 

30.71% 

50.39% 

18.90% 

100% 


Yes 

No 

Yes 

16.67% 

66.67% 

16.67% 

100% 


Yes 

Yes 

No 

51.61% 

45.16% 

3.23% 

100% 


Yes 

Yes 

Yes 

46.96% 

39.02% 

14.02% 

100% 


In the undergraduate portion of the table, we see that the percentages of students who filled out some of 
their evaluations were much greater than on the graduate side. As one undergraduate stated, “1 only did it if teachers 
were really good or really bad. [I was] too lazy to do surveys for mediocre teachers.” A student might have missed 
the class in which the professor mentioned the evaluation or he does not regularly check emails. However, when 
their peers in the dorms mention that they filled out their questionnaires, a student could find the site, log in, and fill 
theirs out. The group that did not recall receiving the initial email, but was encouraged by faculty to respond, did 
complete some of their evaluations when they received the reminder. So, for the undergraduates, all three 
reinforcements played a part in their response levels. 

In comparison to the graduate students, undergraduates were interested in responding but did not feel the 
same level of concern or involvement about all the classes they took. Thus, the percentage completing all their 
evaluations is lower for most of the rows than among corresponding graduate groups. 
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Students responding to only some or none of their evaluations were asked to select the reasons that 
corresponded best to why they had not completed all their evaluations. Students were instructed to circle all the 
reasons that applied to them, so many selected more than one reason. The results are displayed in Table 5 and 
divided into those who filled out some of the evaluations and those who did none. We see that on both the graduate 
and undergraduate levels, a very small percentage of students are worried about being identified by the system. The 
emails that they receive clearly state that there will be no identifying data attached to their responses. Anderson et al 
(2006) refer to students who respond as “busy” or “they forgot” as being “disengaged” with the process of 
evaluation. Among the graduate students, these reasons were selected by the largest number of students who 
completed none of the evaluations. Within the undergraduate group, those who completed only some of the 
evaluations also selected ‘busy’ and ‘forgot’ most, but they were also more likely to select ‘busy’ than ‘forgot’. A 
much greater percentage of undergraduates felt that evaluations simply don’t matter, thus, it was a conscious 
decision on their part to not participate. 


Table 5: Reasons for Non-Response (Multiple Answers Allowed) 


Level 


Don’t Matter 

Forgot 

Busy 

Identify Me 

Other 

Graduate 

None 

7.69% 

49.23% 

50.77% 

6.15% 

16.92% 

Graduate 

Some 

4.35% 

41.30% 

39.13% 

6.52% 

23.91% 

Graduate Total 


6.31% 

45.95% 

45.95% 

6.31% 

19.82% 








Undergraduate 

None 

17.12% 

38.74% 

63.06% 

3.60% 

17.12% 

Undergraduate 

Some 

14.38% 

48.75% 

65.63% 

6.25% 

11.88% 

Undergraduate Total 


15.08% 

46.17% 

64.97% 

5.57% 

13.23% 


Among the answers under the choice “other”, many of the written comments indicated that students 
completed evaluations when they felt that the class was either very good or very bad. Otherwise, they did not 
respond. Examples of this type of response were: “Nothing positive or negative to say about the class”, “If the class 
is average, there is nothing to complain about and there is nothing exceptional, then evaluation is just a waste of 
time”, “I only fill out for the profs I really enjoy or really don't enjoy”, and “I had no strong opinions either way 
about some courses”. Another category of written answers indicated that students were not aware of any impact 
their evaluations could have for them or the class. For example, “Most professors don't read or change according to 
what we write”, “Evaluations don’t change any teacher's teaching style”, and “Class is done, I can't take it again, so I 
don't bother evaluating the class”. The third type of other comment that occurred most often concerned the 
evaluation instrument itself. Examples include, “They take too long”, “Surveys were long”, “The electronic ones 
take longer to complete”, and “Way too long!!” 

CONCLUSIONS 

The responses of over a thousand of our students to this brief survey have given us insight into the process 
of online evaluations. The subsequent analyses we performed offered several suggestions about how we might 
increase student engagement in this procedure. To begin with, our studies showed that faculty encouragement had a 
positive effect on both the undergraduate and graduate level in increasing student participation. Furthermore, the 
original email and the reminder email were both noted to be necessary parts of increasing student awareness and 
response. Future efforts, as such, will prompt faculty to promote the importance of completing online evaluations 
(during class time) and the initial email and reminders will continue to be implemented in the evaluation process. 
Finally, based on student comments, future studies in this area will consider ways the questionnaire can be revised 
so that its length (currently 20 multiple choice questions followed by 3 open-ended responses) is not a deterrent in 
student completion of the evaluations. 
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